Rack mounted liquid submersion cooled electronic system

ABSTRACT

A liquid submersion cooling system that is suitable for cooling a number of electronic devices in parallel using a plurality of cases connected to a rack system. The system cools heat-generating components in server computers and other devices that use electronic, heat-generating components and are connected in parallel systems. The system includes a housing having an interior space, a dielectric cooling liquid in the interior space, a heat-generating electronic component disposed within the space and submerged in the dielectric cooling liquid. The rack system contains a manifold system to engage and allow liquid transfer for multiple cases and IO connectors to engage electrically with multiple cases/electronic devices. The rack system can be connected to a pump system for pumping the liquid into and out of the rack, to and from external heat exchangers, heat pumps, or other thermal dissipation/recovery devices.

TECHNICAL FIELD

This disclosure relates to a liquid submersion-cooled electronic arraysystem, and in particular, to a case that is used in a liquidsubmersion-cooled electronic device, for example, a computer server,where many individual logic boards, in individual cases, or logic boardsgrouped together in a single case, can be contained within a racksystem.

BACKGROUND

A significant problem facing the computer industry is heat. The higherthe temperature a component operates at, the more likely it is to fail.Electronics that are operating under high temperature conditions have ashorter life expectancy than components maintained at lower operatingtemperatures. Generally, it holds true that the higher the temperatureof operation, the shorter the component life expectancy. Also, hightemperatures, while not causing catastrophic failures, can create dataprocessing errors. Operation at high temperatures can cause powerfluctuations that lead to these errors within a central processing unit(CPU) or on the logic board anywhere that data management is handled.Despite efforts at reducing waste heat while increasing processingpower, each new CPU and graphics processing unit (GPU) released on themarket runs hotter than the last. Power supply and logic boardcomponents required to provide power and handle signal processing alsoare producing more and more heat with every new generation.

The use of liquids in cooling systems to cool computer systems is known.One known method of cooling computer components employs a closed-loop,2-phase system. The vapor travels through a tube to a cooling chamber,the vapor turns back into liquid, and the liquid is returned by tube tothe chips for further cooling. In another known liquid cooling system,internal pumps move liquid past a hot plate on a CPU and then the heatedliquid is pumped into a finned tower that passively cools the liquid andreturns it to the plate.

In the case of large-scale, fixed-installation supercomputers, it isknown to submerge the active processing components of the supercomputerin inert, dielectric fluid. The fluid is typically allowed to flowthrough the active components and then it is pumped to external heatexchangers where the fluid is cooled before being returned to the mainchamber.

Despite prior attempts to cool computer components, further improvementsto cooling systems are necessary.

SUMMARY

An individually contained liquid submersion-cooled system is describedthat is suitable for cooling a number of electronic devices, includingcooling heat-generating components in computer systems and other systemsthat use electronic, heat-generating components. Examples of electronicdevices to which the concepts described herein can be applied include,but are not limited to: servers including blade servers; diskarrays/storage systems; storage area networks; network attached storage;storage communication systems; work stations; routers; telecommunicationinfrastructure/switches; wired, optical and wireless communicationdevices; cell processor devices; printers; power supplies; displays;optical devices; instrumentation systems, including hand-held systems;military electronics; etc.

The electronic device can include a case having an interior space. Adielectric cooling liquid is contained in the interior space, and aheat-generating electronic component is disposed within the space and issubmerged in the dielectric cooling liquid or dielectric cooling liquidis directed over the component.

When the electronic device is a computer, for example, a servercomputer, a single logic board or a plurality of logic boards aredisposed within the interior space. The logic board(s) includes a numberof heat-generating electronic components, including at least oneprocessor, for example, a CPU or a GPU. In addition, otherheat-generating components of the computer can be submerged in thecooling liquid, for example, RAM, power supply, daughter cards andstorage drives such as solid-state drives or mechanical hard drives.

In one embodiment, the electronic components need not actually besubmerged in the cooling liquid. Instead, the cooling liquid can be“poured” or otherwise directed over the electronic component(s), withgravity assisting the liquid to flow downward over the component(s),with the liquid thereafter being collected in a sump where it is pumpedto a thermal dissipation/recovery device for eventual return back to theelectronic component(s). This embodiment would reduce the amount ofcooling liquid within the case, thereby reducing weight and cost.

The case containing the computer logic board(s), daughter cards, powersupplies and other active electronic components includes a plurality ofwalls defining a liquid-tight interior space. If desired, one or more ofthe walls can be a transparent, a translucent, or an opaque material. Alid, which can be removable or fixed, closes the interior space, forexample the top of the space. The lid forms a liquid-tight seal with theplurality of walls, and in one embodiment the lid includes a sealedelectrical connector fixed thereto that is configured to attach to thelogic board disposed in the interior space and to provide electricalconnection between the logic board and an exterior of the case.

In an embodiment, when the logic board is lifted from the interiorspace, a mechanism can also be provided to hold the logic board in itsraised position for changing out logic board components and allowingliquid to drain back into the interior space.

The aforementioned case with the aforementioned interior space may alignwith other cases with similar interior spaces to form an array of casesthat can be used for logic boards on servers, storage systems (includingdisk drives), routers, communications devices, and other electronicdevices.

DRAWINGS

FIG. 1 shows an array of cases integrated with an electronic system anda cooling system.

FIG. 2 shows an individual case with the removable lid in an openposition.

FIG. 3 shows the rack system that holds the array of cases.

FIG. 4 shows the case with the logic board, video/daughter board, powersupply unit, top plane board and an external input/output connector thatengages with the rack input/output bus.

FIG. 5 shows a case filled with dielectric fluid and quick-disconnectvalves that engage the rack manifold quick-disconnect valves.

FIG. 6 shows a cooling system including the array of cases, an externalpump and a thermal dissipation/recovery device.

FIG. 7 is a side view of an alternative embodiment of a case that isusable in an array.

FIG. 8 is a side view of yet another embodiment of a case.

FIG. 9 is a side view of still another embodiment of a case.

FIG. 10 is a front view of a case with an ergonomic handle.

FIG. 11 is a front view of a vertically stacked array of cases.

FIGS. 12-14 illustrate side views of cases that contain different typesof heat generating electronics.

FIG. 15 illustrates an exemplary coolant distribution system that can beused with the array of cases.

FIG. 16 illustrates the use of a fluid cooling loop to cool thedielectric cooling liquid.

FIG. 17 is a front view of another embodiment of a case in which aplurality of logic boards are contained.

FIG. 18 is front view of another embodiment of a rack system in whichthe cases are oriented horizontally.

FIGS. 19A, 19B and 19C are an isometric view, a top view with the topwall removed, and a side view, respectively, of a directed flow heatsink and dispersion plenum assembly for cooling a targeted hot spot on alogic board.

DETAILED DESCRIPTION

A liquid submersion-cooled system is described that is suitable forcooling a number of array-based electronic devices, including coolingheat-generating components in computer server systems and other systemsthat use electronic, heat-generating components. In the case of computerserver systems, the liquid submersion-cooled system permits creation of,for example, arrays of computers with scalable architectures where it ispossible to produce 32 to 64, or more, processor core logic boards (8sockets×8 cores=64 processor). These logic boards in a case system,forming an array, can be interconnected to form one or moremassively-parallel super computers.

Examples of electronic devices to which the concepts described hereincan be applied include, but are not limited to: servers including bladeservers; disk arrays/storage systems; storage area networks; networkattached storage; storage communication systems; work stations; routers;telecommunication infrastructure/switches; wired, optical and wirelesscommunication devices; cell processor devices; printers; power supplies;displays; optical devices; instrumentation systems including hand-heldsystems; military electronics; etc. Many of the concepts will bedescribed and illustrated herein as applied to an array of computerservers. However, it is to be realized that the concepts describedherein could be used on other electronic devices as well.

FIGS. 1, 2, 3, 4, and 5 illustrate one embodiment of an array-basedserver computer system 10 employing a liquid submersion cooling system.The system 10 includes a plurality of separate cases 12 each containinga self-contained computer system, for example, a computer logic board,daughter cards, power supplies and other active electronic computercomponents, and a cooling system.

All electronic and thermally active components of each case 12 aresubmerged in dielectric cooling liquid where the dielectric coolingliquid is in direct contact with the electronically and thermally activecomponents. Thus, each case forms a tank containing the cooling liquidand the computer system. Dielectric liquids that can be used in thistype of immersive cooling system include, but are not limited to:

-   -   Engineered fluids like 3M™ Novec™    -   Mineral oil    -   Silicone oil    -   Natural ester-based oils, including soybean-based oils    -   Synthetic ester-based oils        Many of these dielectric fluids also have the ability to        extinguish fires on computer components. By submerging computer        components in a dielectric, fire-retardant fluid, the chance of        a fire starting due to computer component failure is minimized.        Other dielectric liquids that have a higher boiling temperature        along with greater thermal transfer capability can be employed.        These cooling liquids need not change state if they have a high        enough thermal transfer capability to handle the amount of heat        being generated by components contained in the system.

With reference to FIGS. 2 and 4, each case 12 includes a lid 14 thatattaches to the connector side of a computer logic 16, allowing logicboard input/output (IO) connections, daughter card IO and power to bepassed in and out of the case 12. Components such as daughter cards,additional processors, power supply card, and memory cards, which mountonto the logic board 16, can be added to or removed from the case 12 byopening the case lid 14 and lifting the attached logic board andelectronics out of the case. The edges of the logic board 16 can beslideably disposed inside grooves formed on opposite inside walls of thecase 12 to facilitate insertion and removal of the logic board from thecase. In addition, a mechanical hard drive can be disposed in the case12, with an air line connected to the hard drive breather hole leadingfrom the hard drive to the exterior of the case.

In the illustrated example in FIGS. 2, 4, and 5, the rear of the case 12(when the case is oriented during use) is closed by the lid 14. Toaccess the interior of the case 12, the case is oriented handle end downso that the lid 14 is disposed at the top. The lid 14 can then beremoved to lift the attached logic board 16 out of the case. Otherorientations of the lid 14 are possible. For example, the lid couldclose the top of the case 12 (when the case is oriented during use) orclose any other side of the case.

In one embodiment, an internal or external pumping system can be used topump warm liquid from the top of the cases and pass it through externalheat exchangers, heat pumps, or other thermal dissipation/recoverydevices. In other embodiments, the flow can be convective, or resultfrom gravity, avoiding the need for a pump(s). The flow of liquid canalso be caused by various combinations of pumps, convection, andgravity.

An exemplary embodiment is illustrated in FIG. 6 which shows the cases12 arranged in an array, with the cases connected to an outlet manifold20, an external pump 22 that pumps liquid from the cases 12 to a thermaldissipation or recovery device 24, with the cooled liquid then beingdirected to an inlet manifold 26 for directing the cooled liquid backinto the cases.

If desired, more than one thermal dissipation/recovery device can beutilized, and a combination of thermal dissipation and thermal recoverycan be used. In addition, a pump can be disposed inside each case 12 ifdesired.

FIG. 15 illustrates another example of a coolant distribution system 30connected to the array of cases 12 for distributing coolant liquid tothe cases and receiving coolant from the cases 12. The illustratedsystem 30 includes a thermal dissipation or recovery device 32, acoolant supply line 34, and a coolant return line 36. The coolant wouldbe pumped by a suitable pump that is part of the thermal dissipation orrecovery device 32 or by an external pump similar to the system in FIG.6. The system 30 could be, for example, a two-pipe reverse return systemwhere the first case 12 that is supplied with coolant liquid is the lastto return coolant, so that the flow path of coolant liquid to and fromeach case is the same length. An advantage of a two-pipe reverse-returnsystem is that it equalizes the pressure drop across the cases.

The thermal dissipation or recovery device can be any device that issuitable for dissipating heat or allowing recovery of the heat from theheat liquid. For example, the device can be a simple heat exchanger,such as a radiator, for dissipating heat. Air or liquid could be used asthe heat exchanging medium. In addition, the heat exchanger could bedisposed underground to allow the relatively cool ground to cool theliquid. The external heat exchanger can take on a number of differentconfigurations, as long as it is able to cool the liquid down to anacceptable temperature prior to being fed back into the space. Examplesof thermal dissipation devices include, but are not limited to, acooling stack, evaporation, and an in-ground loop.

Cooling of liquids utilizing a heat exchanger can be accomplished by oneof several means:

-   -   A compressor, as is the case with typical refrigeration systems    -   Peltier effect cooling    -   Active air cooling of the radiator surface using a fan or other        air-moving mechanism    -   Passive cooling by exposing as large of a thermally conductive        heat exchange surface as possible to lower ambient temperatures

The thermal dissipation or recovery device could also be a heat recoverydevice where recovered heat is used for environmental heating. Forexample, the heat recovery device can be part of a building or roomheating system where recovered heat is used to heat the building.Examples of thermal recovery devices include, but are not limited to,in-floor heaters and geothermal electricity generation.

In one embodiment, the cases are contained in a room while the pump andthe thermal dissipation or recovery device are located outside the room.Because liquid is used for cooling and the heated liquid is pumpedoutside the room, heating of the room by the electronics in the cases isminimized. This reduces the amount of air conditioning that is requiredinside the room, which reduces electrical use and the cost ofmaintaining the array of cases in the room.

The dielectric cooling liquid need not be the only liquid used in thethermal dissipation or recovery system. In the embodiment illustrated inFIG. 16, a water cooling loop 40 is used to cool the dielectric coolantof the cases 12. The dielectric coolant circulates in a loop through aheat exchanger 42. The water cooling loop 40 also runs through the heatexchanger 42 where it exchanges heat with and cools the dielectriccoolant. The water in the loop 40, which can circulate via a pump 44,convection, and/or by gravity, is then directed to a thermal dissipationor recovery device 46.

The cases 12 will be connected in an array using a rack system 50 asshown in FIG. 3 and will connect to inlet and outlet manifolds via quickdisconnect valves 52 disposed at the rear of the cases 12 as shown inFIGS. 2 and 5. One valve 52 will act as an inlet 52 a for coolant intothe case, while the other valve 52 will act as an outlet 52 b forcoolant from the case, with the valves preventing flow of liquid throughthe inlet 52 a and outlet 52 b unless the valves are opened. The valves52 will engage inlet 53 a and outlet 53 b manifolds disposed at the rearof the rack system 50, shown in FIG. 3. The flow lines of the inlet andoutlet manifolds 53 a, 53 b leading to the cases 12 will also utilizequick-disconnect valves that engage with the valves 52, for example sothat the valves 52 on the case and the valves on the manifolds 53 a, 53b will automatically open upon installation of a case, and automaticallyclose upon removal of the case. The quick-disconnect valves would allowthe change-out of a failed server computer using a hot swappable systemarchitecture.

The rack system 50 includes a frame 54, a coolant return line 56connected to the inlet manifolds 53 a at the rear of the frame 54, and acoolant outlet line 58 connected to the outlet manifolds 53 b. The cases12 are each mountable on the frame 54 to support the cases in thedesired array configuration. The frame 54 illustrated in FIG. 3 isillustrated as being capable of supporting three vertically-spaced rowsof cases that are slid into the frame 54. Each case 12 can be providedwith means to facilitate sliding insertion and removal of the case. Forexample, a bearing mechanism can be provided on the base of each casethat makes it easier for the cases to slide into and out of the frame54.

FIGS. 2 and 4 show a number of active computer components mounted on thelogic board that require cooling, including a video/daughter board 62,power supplies 64, and processors (disposed under the dispersion plenums74). In some embodiments, a pump can also be mounted on the logic board16 or on the interior of the lid 14. A top plane board 70 is connectedto the underside of the lid 14, and an external input/output connector72 is disposed at the rear of the lid 14. The external I/O connector 72engages with an input/output bus 73 on the rack system 50 shown in FIG.3. Each case 12 is large enough to contain all of the active computercomponents that require cooling. It may also be necessary to leave spacefor liquid return lines and/or dispersion plenums 74 that are used todirect cooling liquid over specific, high-temperature areas, for examplethe CPUs, as discussed below with respect to FIG. 7.

As shown in FIGS. 2 and 4, the lid 14 not only provides a liquid- andgas-tight seal for the case 12, but it also contains the pass-throughconnector 72 that allows external component IO, storage IO and power topass into and out of the case 12 to and from the computer logic board 16and its components. The lid 14 has a gasket that will seal the case. Thelid 14 may also contain a fill port for filling the case with coolant.

With reference to FIG. 4, the server logic board 16 is essentiallyfunctionally the same as in current server specification boards, withthe exception being that it does not have the same IO and powerconnectors. Instead, the top edge of the logic board 16 can be providedwith, for example, a series of conductive pads that are contacts forengaging the pass-through connector 72. Alternatively, a liquid-tightbus-type pass-through connector may be employed to pass IO and powerthrough the case 12.

Multiple logic boards or other circuit boards may be employed to allowstacking of extra processors or other components for additionalcomputing power or to allow for multiple computers within a single case.For example, FIG. 17 illustrates a case 500 that is constructed similarto the case 12, but containing a plurality of logic boards 501.

The described cooling system would allow for numerous server computersystems to be cooled in a single case (FIG. 17) or individual caseswhich may be interconnected to create a server or workstation racksystem (FIGS. 1-6).

With reference to FIG. 4, the daughter card(s) 62 connect to the logicboard 16 as they do with current server specification boards. Daughtercards can include video cards and other PCI or PCIE cards that requireIO pass-through to the outside of the case. These daughter cards willrequire liquid- and gas-tight gaskets in order to allow external IOconnections.

Unlike current server designs, the power supplies 64 may also bedaughter cards, with no power supply to logic board wiring required. Thepower supplies may also be directly integrated into the logic board 16.External alternating current (AC) or direct current (DC) connectionswould be made through the pass-through connector 72 into theliquid-filled case with a liquid and gas-tight gasket.

The pumping system, if used, is preferably externally mounted,supporting all of the cases 12 in the array. The pumping system is usedto circulate warm liquid from inside the cases 12 to outside of thecases to the thermal dissipation/recovery device(s). Liquid may also becirculated through external hard drive cooling plates as well. Thepumping system can be wired such that it can be turned on to circulateliquid even if the server computers are turned off. Or the pumpingsystem can be wired to turn on only when the server computers are on.After the server computers are shut off, there is more than sufficientthermal capacity in the liquid within the cases 12 to remove residualheat from the submerged components. This would ensure that there is nopost-shut down thermal damage. Also, if a flow sensor or pump monitorindicates that flow of coolant has stopped or has slowed below a minimumrequired rate, a controlled shutdown of the server computers could becompleted well before any damage is done to the submerged components.This embodiment avoids the possibility of a pump system failure,resulting in catastrophic failure of a server computer that relies onair cooling.

Hard drives or other internal storage systems can also be submerged. Inthe case of current platter-based, mechanical storage systems thatrequire breather holes, the air line could be fixed over the breatherhole, allowing an open-air connection to the outside of the case. Therest of the drive would be sealed as to be gas and liquid impermeable.

The processors mount to the logic board 16 via normal, vender-specifiedsockets. Testing has shown that, in some instances, no heat sinks orother appliances need to be attached to the processors in order to coolthem sufficiently for normal, vendor-specified temperatures. However, iflower operating temperatures or a higher level of heat transfer isrequired for higher-powered processors or processor over-clocking, heatsinks, which greatly increase the exposed surface area of heatconduction from the processor(s), may be employed.

When a case 12 is removed from the array rack 50, electrical power (ACor DC) becomes disengaged from the external connector 72 of the case 12,opening the electrical circuit and disconnecting electrical power (AC orDC) from the interior space of the case 12.

The lid 14 can also include an opening through which cooling liquid canbe added into the space. The opening is closed by a removable cap whichis removed when liquid is to be added. The lid can also include a lockmechanism that locks the lid in place and locks the case into the racksystem 50 as shown in FIG. 3.

As described above, the server logic board assembly is removable anddisposed in the interior space to permit the server logic board assemblyto be lifted from the space when the lid is lifted upward.

The interior space of each case 12 should contain enough dielectriccooling liquid to submerge the components that one wishes to besubmerged. For example, the cooling liquid may substantially fill theinterior space, whereby all heat-generating components on the logicboard 16 are submerged. The cooling system is designed to direct heateddielectric liquid from inside the space and through the externalquick-disconnect valves 52 to the heat exchanger(s), heat pump(s), orother thermal disposition/recovery devices where the liquid is cooled.The cooled liquid is then returned to each case through the rack system50 and the inlet 52 a and outlet 52 b.

FIG. 7 illustrates a case 100 that is usable in an array. The case 100is designed to electrically connect to a backplane IO board 102. Thecase 100 includes a liquid-tight housing 104 inside of which is acircuit board 106, for example a logic board, mounting various heatgenerating electrical components that are submerged in cooling liquid. Alid 108 closes the rear of the housing 104. The lid 108 can be removableto allow access to the interior of the housing, or the lid can benon-removable to permanently close the housing.

A liquid inlet 110 is formed on the back side of the housing near thebase thereof and a liquid outlet 112 extends from the back side adjacentthe top. This arrangement of the inlet and outlet introduces the cooledliquid near the bottom, and since the liquid will rise as it is heated,the heated liquid can exit the outlet 112. The inlet 110 and outlet 112are provided with quick-connect valves, where the valves are designed toautomatically open upon connection with a manifold inlet 120 and amanifold outlet 122, respectively, that extend through the backplaneboard 102. The inlet 120 and outlet 122 are also provided with quickconnect valves that automatically open upon connection, with the inlet120 leading from an inlet manifold (see FIG. 6) and the outlet 122connecting to an outlet manifold (FIG. 6).

An IO connector 114 extends from the back side of the housing, with theconnector 114 extending through the back wall of the housing and beingconnected to the circuit board 106. The IO connector 114 is designed toelectrically connect with a connector 116 of the backplane IO board 102to direct inputs and outputs to and from the circuit board, and ifnecessary to pass electrical power into the housing. A similar connector116 is provided to connect to the connector 114 of each case 100 of thearray.

In areas where there is significant heat, directed liquid flow can beused to provide localized cooling. In particular, as shown in FIG. 7, adirected liquid cooling assembly 130 is connected to the inlet port. Theassembly 130 includes a manifold 132 in communication with the inlet110, and a plurality of tubes 134 that extend from the manifold todirect liquid to specific targeted hot spots, such as CPUs, on thecircuit board 106. The ends of the tubes 134 are positioned adjacent thedesired hot spots, or the tubes 134 can connect to dispersion plenums136, similar to the dispersion plenums 74, to help direct the flow ofthe return liquid to the targeted hot spots.

FIGS. 19A-C illustrate details of one example of a directed flow heatsink and dispersion plenum assembly 700. The assembly 700 includes adispersion plenum housing 702 that is a structure made of a suitablematerial, for example plastic or metal, having perimeter side walls 710,and a top wall 712. An inlet tube 701, similar to the tube 134, connectsto an inlet port 714 provided on one of the perimeter side walls 710 fordirecting coolant into the interior of the housing 702. Although asingle inlet port 714 is illustrated, a plurality of inlet ports can beprovided, each connected to an inlet tube 701. The number of inlet portsand the tubes connected to the inlet ports can vary depending upon thecooling needs of the targeted hot spot.

The housing 702 closely envelopes a specific targeted hot spot, such asa GPU or a CPU 706, that has a heat sink 703 mounted thereto. The CPU706 is mounted on a socket 707 on the circuit board 708. The heat sink703 can comprise, for example, a plurality of fins 704 that extendupwardly from engagement with the CPU 706 toward the top wall 712. Thefins 704 include substantial surface area to optimize heat exchange withthe CPU. The heat sink fins may be machined, skived, cast or otherwiseformed to create a large surface area for heat removal from the CPUthrough direct contact with the cooling liquid. In the illustratedexample, the fins 704 are oriented generally parallel to the coolantflow direction shown by the arrows. To help mix or break up the flow,gaps 705 can be provided between the fins 704.

An opening 718 is formed in one side of the housing 702 to allow coolantto flow out of the interior space. The opening 718 can be provided atany location that allows escape of the coolant from the interior of thehousing 702 as the coolant is heated through heat exchange with the finsand the CPU. The heated coolant may rise upwardly toward the top of thecase after leaving the housing 702 or be expelled in any direction thatthe opening 718 is facing due to pressure created by the pump supplyingthe plenum assembly. In the illustrated embodiment, the opening isformed by a gap formed in the perimeter side wall 710 opposite the inletport 714. More than one opening can be provided, and the opening can beprovided at any suitable location on the housing 702 to allow the heatedcoolant to escape the housing 702 and rise upward toward the top of thecase or be ejected in the forced flow direction.

In addition, the housing 702 defines an expansion chamber 716 near theinlet port 714 that allows the coolant entering the housing to flow inmultiple directions so as to flow over the entire expanse of the fins704.

In use, directed flow through the tube 701 from the manifold, or flowdirectly from a pumped source, is contained within the confined spacedefined by the plenum housing 702. This confined space closely envelopesthe CPU and the heat sink fins 704 connected to the CPU. The plenumhousing 702 contains the liquid flow, forcing the liquid to have mixedflow or, under certain conditions turbulent flow, throughout theentirety of the heat sink surface. This helps to optimize heat exchangein order to more efficiently cool hot components, such as CPUs, GPUs,north bridge, south bridge and/or other components that create asubstantial amount of heat relative to the rest of the electronicssystem in which they are utilized. After contacting the CPU and fins,the liquid exits the plenum housing 702 through the opening 718.

The plenum housing 702 need not form a liquid-sealed enclosure over theCPU and fins, and in the illustrated example does not form aliquid-sealed enclosure. This helps eliminate the need for tightmanufacturing tolerances. The plenum housing 702 should be configured tohelp contain the liquid as it flows over the CPU and fins. The amount ordegree of flow containment can vary depending upon how much flowcontainment, and resulting increase in cooling effectiveness, one wishesto achieve.

The case 100 is designed to slide into a space between upper and lowershelves 140 a, 140 b of the rack. Other cases of similar design would bedisposed between the shelves 140 a, 140 b next to the case 100. Tofacilitate insertion and removal of the case 100, a handle 142 can beformed on the housing 104. In addition, hinged locks 144 a, 144 b can beprovided on the case 100 and/or on the shelves 140, 140 b to hold thecase 100 in place in the rack.

FIG. 8 illustrates a case 200 that is designed to achieve electrical andliquid connection via a vertical upward movement of the case. The case200 is generally similar to the case 100 in FIG. 7, including aliquid-tight housing inside of which is a circuit board mounting variousheat generating electrical components that are submerged in coolingliquid. The case 200 is designed to connect with a top plane 202 mountedon an upper shelf 204. The top plane 202 has an IO connector 206 thatconnects with an IO connector 208 that extends from the top of the casehousing, with the connector 208 extending through the top of the housingand being connected to the circuit board inside the housing.

In addition, a liquid inlet 210 and a liquid outlet 212 are formed onthe bottom of the housing. The inlet 210 and outlet 212 are providedwith quick-connect valves, where the valves are designed toautomatically open upon connection with a manifold inlet 214 and amanifold outlet 216, respectively, that extend from and through avertically movable base 220. The inlet 214 and outlet 216 are alsoprovided with quick-connect valves that automatically open uponconnection.

A lever system is engaged with the base 220 for actuating the basevertically. The lever system includes a lever 224 that is pivotallyconnected adjacent the base 220 and engaged therewith such that when thelever 224 is pivoted it actuates the base upwardly. When the base movesupward, fluid engagement between the inlet and outlet 210, 212 and themanifold inlet and outlet 214, 216 is achieved, and electricalconnection between the IO connectors 206, 208 is achieved.

FIG. 9 illustrates a case 300 that is similar to the case 100 in FIG. 7,including liquid inlets 302 and outlets 304 on the back side of the casehousing that connect to liquid inlet 317 and outlet 318 lines,respectively. A backplane IO board 306 includes an IO connector 308. Atop plane 310 is connected to the top of the case 300 having an IOconnector 312 connected to an IO connector 314 extending through the topof the case housing and engaged with the circuit board in the casehousing. The top plane 310 includes an attached flex portion 316containing an IO connector 318 that is designed to connect to the IOconnector 308.

The case 300 can also be mounted on a telescoping bearing slide 320which allows the case to slide in and out of position. The slide 320 isfixed to the lower shelf 322 and includes a first slide portion 324 anda second slide portion 326 slideable relative to the first slide portion324.

FIG. 10 illustrates the front of a possible case design where the casehousing 350 has side walls that slope outwardly from front to back sothat the housing is wider at the back than at the front. In addition,the front of the housing from top to bottom is curved in a crescentshape to form a generally hour-glass shaped handle 352 which can begripped to facilitate insertion and removal of the case from the rack.

FIG. 11 illustrates an array 400 of cases 402 stacked in vertical levels404 a, 404 b, 404 c. Each level 404 a-c includes a plurality of cases402 arranged side-by-side. A liquid intake storage tank 406 is disposedat the top of the array 400 for gravity feeding liquid to the cases 402.A heat dissipation device 408 is provided at the bottom of the array foruse in cooling the liquid after it is output from the cases. The heatdissipation device 408 can include a radiator through which the coolingliquid passes, and a fan for blowing air across the radiator for coolingthe liquid.

The arrangement in FIG. 11 utilizes gravity to feed cooling liquid tothe individual cases, and to discharge liquid from the cases down to theheat dissipation device 408. A pump 410 is provided to pump the liquidback up to the intake storage tank 406, or pump the liquid to anexternal heat dissipation device prior to being delivered to the intakestorage tank 406. An optional pump 412 can also be provided for pumpingthe liquid to the cases from the tank 406 rather than relying solely ongravity.

Instead of arranging the cases side-by-side (which can be referred to asa horizontal array) in a plurality of vertical levels as illustrated inFIG. 11, the individual cases can instead be oriented horizontally, withthe cases stacked one above the other, which will be referred to as avertical array. For example, FIG. 18 illustrates an array 600 thatincludes a plurality of horizontally-oriented cases 602 disposed oneabove the other in a rack 601. The cases 602 could be in abuttingcontact with adjacent cases, or gaps can be provided between each case.Further, the cases 602 could be separated into discrete vertical levelssimilar to the vertical levels described in FIG. 11.

The embodiments described above referenced server cases containingserver electronics. However, the concepts described herein can beapplied to other electronic arrays containing other heat generatingelectronics. For example, FIG. 12 illustrates a power or battery case450 containing a plurality of power sources 452. A plurality of thecases 450 arranged in an array as described herein can providesignificant power resources in a relatively compact space. FIG. 13illustrates a case 460 that can contain devices 462 such as a pluralityof storage area network devices or network attached storage devices, aplurality of routers, or communication electronics. This would permitterabytes, or more, of data to be put on flash storage on a singleboard. FIG. 14 illustrates a case 470 where one or more power supplyunits 472 are disposed on a board, along with a plurality of devices 474such as processors or memory. Since the power supply unit 472 is insidethe case, this eliminates the need to pass DC power from an externalsource into the case. AC power could be passed into the case and therequired DC power would be converted inside the case.

The dielectric liquid that is used as the cooling liquid can be any ofthe dielectric liquids discussed above. If desired, a colorant materialcan be added to the dielectric liquid to make the liquid a particularcolor. The colorant may also be ultraviolet light (UV)-activated. Thiscould be used in conjunction with UV lights to make the fluid glow aparticular color.

IO connections can be any type of IO connection by which electricalinputs and/or outputs are passed into and/or out of the cases. Examplesof IO connections include, but are not limited to, fiber channel,Ethernet, serial attached SCSI (SAS), serial advanced technologyattachment (SATA), USB, video, wireless (WIFI, RF network), temperatureor other environmental monitoring.

The embodiments disclosed in this application are to be considered inall respects as illustrative and not limitative. The scope of theinvention is indicated by the appended claims rather than by theforegoing description; and all changes which come within the meaning andrange of equivalency of the claims are intended to be embraced therein.

The invention claimed is:
 1. A system, comprising: a rack configured forreceiving a plurality of electronic systems thereon, the rack having aninlet manifold and an outlet manifold; a plurality of electronic systemsmounted on the rack in a horizontal array, each electronic systemcomprising: a sealed case defining a liquid tight interior space, afluid inlet in fluid communication with the interior space and fluidlyconnected to the inlet manifold, a fluid outlet in fluid communicationwith the interior space and fluidly connected to the outlet manifold,and a plurality of heat generating electronic devices disposed withinthe interior space; a single-phase, dielectric cooling liquid disposedwithin the interior space and submerging the heat generating electronicdevices; a heat exchanger mounted on the rack, the heat exchangerincluding an inlet in fluid communication with the outlet manifold andan outlet in fluid communication with the inlet manifold; a pump fluidlyconnected to the heat exchanger for pumping the single-phase, dielectriccooling liquid through the heat exchanger; and a liquid cooling systemin heat exchange relationship with the heat exchanger.
 2. The system ofclaim 1, wherein the heat generating electronic devices compriseprocessors, switches, power supplies, network electronics, or datastorage or memory devices.
 3. The system of claim 1, wherein theelectronic systems comprise servers, routers, switch systems,telecommunication networks, power supply systems, or data storage ormemory systems.
 4. The system of claim 2, further comprising a pluralityof boards disposed within the interior space that are arranged parallelto and spaced from one another, and the heat generating electronicdevices comprise a plurality of the data storage or memory devicesmounted on each board.
 5. The system of claim 2, further comprising aplurality of boards disposed within the interior space that are arrangedparallel to and spaced from one another, and the heat generatingelectronic devices comprise a plurality of the switches mounted on eachboard.
 6. The system of claim 2, wherein the heat generating electronicdevices comprise a plurality of power sources.
 7. The system of claim 1,wherein the fluid inlet includes a quick connect valve, and the fluidoutlet includes a quick connect valve.
 8. The system of claim 1, furthercomprising a manifold in the interior space of the sealed case andconnected to the fluid inlet, the manifold includes a plurality ofmanifold outlets.
 9. The system of claim 8, further comprising aplurality of tubes, each tube having one end connected to a respectiveone of the manifold outlets and a second end positioned relative to anassociated one of the heat generating electronic devices to directincoming flow of single-phase dielectric cooling liquid from the fluidinlet and the manifold onto the associated heat generating electronicdevice.
 10. A system, comprising: a rack configured for receiving aplurality of electronic systems thereon, the rack having an inletmanifold and an outlet manifold; a plurality of electronic systemsmounted on the rack in a side-by-side horizontal array, each electronicsystem comprising: a sealed case defining a liquid tight interior space,a fluid inlet in fluid communication with the interior space and fluidlyconnected to the inlet manifold, a fluid outlet in fluid communicationwith the interior space and fluidly connected to the outlet manifold,and a plurality of heat generating electronic devices disposed withinthe interior space; a single-phase, dielectric cooling liquid disposedwithin the interior space and submerging the heat generating electronicdevices; a thermal dissipation or recovery device including an inlet influid communication with the outlet manifold and an outlet in fluidcommunication with the inlet manifold; and a pump fluidly connected tothe thermal dissipation or recovery device for pumping the single-phase,dielectric cooling liquid through the thermal dissipation or recoverydevice.
 11. The system of claim 10, wherein the thermal dissipation orrecovery device is a heat exchanger or a heat recovery device.
 12. Thesystem of claim 10, wherein the heat generating electronic devicescomprise processors, switches, power supplies, network electronics, ordata storage or memory devices.
 13. The system of claim 10, wherein theelectronic systems comprise servers, routers, switch systems,telecommunication networks, power supply systems, or data storage ormemory systems.
 14. The system of claim 10, wherein the rack is disposedin a first room, and the thermal dissipation or recovery device and thepump are disposed outside the first MOM.
 15. The system of claim 10,wherein the fluid inlet includes a quick connect valve, and the fluidoutlet includes a quick connect valve.
 16. The system of claim 10,further comprising a manifold in the interior space of the sealed caseand connected to the fluid inlet, the manifold includes a plurality ofmanifold outlets.
 17. The system of claim 16, further comprising aplurality of tubes, each tube having one end connected to a respectiveone of the manifold outlets and a second end positioned relative to anassociated one of the heat generating electronic devices to directincoming flow of single-phase dielectric cooling liquid from the fluidinlet and the manifold onto the associated heat generating electronicdevice.